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TEST  PLANNING 


2.1  BNTOODUCnON 

We  must  never  make  test  and  evaluation  (T&E)  an  end  unto  itself.  T&E  is  a  subset  of 
engineering.  Regardless  of  who  conducts  the  tests,  makes  the  evaluations,  and  how  the  test 
results  are  used  for  program  approval,  T&E  must  be  structured  to  take  its  input  from  and 
give  its  output  to  engineering.  It  must  be  tailored  to  the  technical  issues  of  the  particular 
endeavor  it  is  supporting  whether  it  be  an  R&D  experiment,  an  operational  test,  a  first 
article  test  or  even  an  in-house  maintenance  test.  Our  customers,  the  Defense  establishment, 
and  the  public  cannot  afford  T&E  programs  that  are  fully  standardized,  stand-alone  and  are 
planned  to  prove  everything.  T&E  must  be  judiciously  planned  to  ensure  it  is  efficient  and 
cost  effective. 

The  following  sections  will  introduce  the  acquisition  process  and  discuss  test  planning 
from  both  a  mechanical  and  philosophical  approach.  The  final  section  outlines  the  mechanics  of 
planning  the  USAF  Test  Pilot  School  test  management  project.  The  ^pendicles  contain 
documents,  forms,  and  samples  to  simplify  the  TPS  test  management  projects  as  well  as 
instruct. 

2.2  AIRFORCETESTANDEVALUATIONPOLICY  BACKGROUND 

The  test  and  evaluation  (T&E)  of  aircraft  and  air  weapons  systems  can  be  traced 
directly  to  the  contract  awarded  the  Wright  brothers  in  1908.  This  document  specified  a 
craft  capable  of  lifting  two  men  with  a  total  weight  of  350  pounds,  carrying  enough  fuel  for 
a  flight  of  125  miles,  and  flying  40  mph  in  still  air.  It  outlined  tests  to  assure  this  capability. 

In  1934,  the  Baker  Committee  (formed  to  study  aU  aspects  of  the  Air  Corps) 
recommended  that  a  separate  branch  be  established  for  research  and  flight  testing.  In  1941 , 
helped  by  the  availability  of  the  Choctawhatchee  National  Forest,  an  air  proving  ground  was 
established  in  the  area  that  is  now  Eglin  Air  Force  Base.  Disputes  over  the  jurisdiction  of  the 
area  with  the  Department  of  Interior  delayed  rapid  progress  of  the  Air  Proving  Ground  until 
the  spring  of  1942.  Progress  was  further  delayed  as  World  War  n  focused  attention  on  total 
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weapons  production  and  the  training,  equipping,  and  deployment  of  combat  forces.  New 
weapons  often  saw  their  first  extensive  tests  in  the  field,  and  concepts  and  doctrine  for 
employment  were  developed  as  a  result  of  the  day’s  losses.  Under  these  conditions,  a  strictly 
disciplined  test  and  evaluation  effort  by  the  Air  Proving  Ground  was  practically  impossible. 

Still,  by  March  1942,  as  a  part  of  the  reorganization  of  the  Army  Air  Corps,  the 
Eglin  organization  was  redesignated  the  Air  Corps  Proving  Ground  Command.  By  1943  it  had 
ordnance  detachments  in  Maryland,  Indiana,  and  Arkansas.  It  also  had  an  electronic  proving 
ground  in  Alabama  and  an  Arctic,  Desert,  and  Tropic  Information  Center. 

At  the  same  time,  the  Army  Air  Forces  School  of  Applied  Tactics  was  forming  to  test 
the  tactical  suitability  of  equipment  the  Proving  Ground  had  found  to  be  operationally  suitable. 
Thus  two  of  the  basic  tasks  of  T&E  were  recognized  and  independent  organizations  were 
established  to  assess  them.  Prior  to  this  time,  nearly  all  T&E  was  performed  at  Wright- 
Patterson  Field  as  part  of  the  procurement  process.  T&E  at  Wright-Patterson  was  necessarily 
oriented  toward  development  and  satisfaction  of  contract  specifications. 

Following  World  War  n  and  the  creation  of  the  U.S.  Air  Force,  the  Air  Proving 
Ground  Command  retained  its  mission  as  an  independent  testing  agency  reporting  directly  to 
the  Air  Force  Chief  of  Staff.  It  drew  its  manning  from  operationally  experienced  people. 

Tests  were  designed  and  executed  in  as  near  the  anticipated  operational  environment  as  could 
be  simulated.  Tactics  development  was  part  of  the  test  when  possible. 

The  Air  Proving  Ground  Command  flourished  and  grew  until  by  the  mid  1950s  it 
consisted  of  approximately  11,000  personnel.  Additional  forces  were  introduced  on  a 
temporary  basis  to  perform  tests.  It  had  an  extensive  inventory  of  the  latest  aircraft  and 
equipment.  However,  much  of  its  efforts  appeared  to  be  diverted  to  its  periodic  fire  power 
demonstrations.  With  this  growth  and  high  visibility,  the  Air  Proving  Ground  Command 
became  vulnerable  to  its  critics.  Charges  that  it  was  too  big,  too  cumbersome,  and  slow  to 
respond  to  developers,  users,  and  the  Chief  of  Staff  brought  a  reorganization  in  1957.  It 
was  reduced  to  Center  status,  4000  personnel  were  reassigned,  and  it  was  made  subordinate 
to  Air  Force  Systems  Command.  It  quickly  lost  its  capability  to  do  effective  independent 
T&E.  Eventually,  the  Air  Proving  Ground  Center  was  combined  with  the  Armament 
Development  and  Test  Center. 
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At  the  same  time,  responsibility  for  the  funding,  planning  and  conducting  of  Air 
Force  operational  T&E  was  transferred  to  using  commands.  No  provision  was  made  for  the 
separate  funding  of  operational  T&E  and  the  function  was  absorbed  into  Operations  and 
Maintenance  funding,  vulnerable  to  every  operational  squeeze  for  manpower  and  money.  Thus 
operational  T&E  became  fragmented  as  each  using  command  pursued  its  own  perceived 
interests. 

Concurrently,  the  Air  Force  developed  a  highly  formalized  testing  procedure  and  related 
it  to  the  weapons  acquisition  and  deployment  schedules  (not  the  other  way  around).  Testing 
was  divided  into  three  separate  categories.  Category  I  tests  were  performed  by  the  contrac¬ 
tor,  often  at  his  own  facility,  and  devoted  primarily  to  verifying  technical  performance  of 
subsystems.  Category  11  testing  was  the  responsibility  of  Air  Force  Systems  Command  and 
was  usually  carried  out  by  a  joint  effort  of  Air  Force  and  contractor  personnel.  Testing  was 
designed  to  validate  technical  performance  against  specification.  The  using  command  and  AFLC 
were  to  monitor  the  testing  in  Category  I  and  II,  but  neither  participated.  Category  III  was 
conducted  by  the  using  command  with  early  production  equipment.  This  gearing  of  category 
testing  to  acquisition  cycle  presented  little  opportunity  for  operational  test  and  evaluation 
results  to  influence  the  production  decision.  During  the  1960s,  the  press  of  supporting 
commitments  in  Southeast  Asia  led  to  a  greater  use  of  concurrency  in  the  acquisition  process. 
This  involved  greater  use  of  high-risk  fixed-free  type  contracts  in  which  development  and 
production  were  defined  in  a  single  effort.  Such  arrangements  called  for  commitments  on 
performance,  schedule,  and  price  before  development  began.  This  was  to  become  known  as 
"total  package  procurement." 

An  inherent  problem  with  the  "total  package"  concept  was  that  weapon  systems  were 
delayed  from  operational  employment  by  performance,  reliability,  and  maintenance  problems 
discovered  after  they  were  deployed.  Delays  in  performance  achievement  often  resulted  in  a 
compression  of  schedule  that  decreased  adequate  development  testing  and  practically  eliminated 
operational  testing. 

By  1970,  system  failures,  high  cost  of  procurement,  and  extensive  post-production 
modifications  began  to  draw  deserved  criticism  from  Congress  and  DOD.  These  agencies  began 
advocating  establishment  of  an  independent  testing  agency  that  could  stop  production  until 
tests  verified  performance  and  reliability  requirements. 
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The  current  trends  in  T&E  can  best  be  traced  to  the  July  1970  Report  of  the  Blue 
Ribbon  Defense  Panel.  The  panel  recommended  that:  (a)  responsibility  for  T«fcE  should  be 
assigned  to  an  Assistant  Secretary  of  Defense  for  Test  and  Evaluation;  (b)  a  separate  funding 
program  element  should  be  established  for  T«feE  and;  (c)  a  Defense  Test  Agency  should  be  set 
up  to  supervise  the  whole  T&E  effort. 

The  General  Accounting  Office  viewed  a  number  of  Air  Force  programs.  They 
concluded  that  most  programs  had  inadequate  test  plans.  T&E  was  not  generally  accomplished 
in  a  timely  and  effective  manner  and  the  value  of  test  reports  was  diminished  by  inadequate 
planning  and  implementation.  More  significant  to  policy  was  their  conclusion  that  complete 
and  adequate  test  data  was  not  available  to  decision  makers  at  key  points  in  the  acquisition 
cycle. 

The  Air  Force  brought  the  matter  under  study  first  by  the  Bolender  Committee,  and  in 
1971,  by  a  Test  Concept  Review  Board.  Both  committees  generally  supported  previous 
criticisms  that  the  operational  test  and  evaluation  program  within  the  Air  Force  could  be 
improved  for  more  responsiveness  to  the  decision  maker’s  needs. 

Congress  got  into  the  picture  in  1972  with  Public  Law  92-156.  This  law  requires  "a 
written  report  regarding  development  and  procurement  schedules  for  each  weapon  system  for 
which  fund  authorization  is  required  and  for  which  any  such  funds  are  requested  in  such 
budget."  Further  it  requires,  "Beginning  with  calendar  year  1973,  there  shall  be  included  in 
the  report  data  on  operational  testing  and  evaluation  for  such  weapon  systems  for  which 
funds  for  procurement  are  requested." 

With  this  congressional  emphasis,  the  Office  of  Management  and  Budget  issued  firm 
policy  (Circular  A- 109,  5  Apr  76)  to  be  followed  by  all  executive  branch  agencies  in  the 
acquisition  of  major  systems.  DOD  implemented  this  policy  with  DODD  5000.1,  DODl 
5000.2,  and  DODD  5000.3. 

DODD  5000.1  details  the  DOD  systems  acquisition  policy.  The  acquisition  cycle  begins 
with  identification  of  an  operational  with  mission  need  (i.e.,  kill  tanks  at  night,  defend  the 
east  and  west  coasts  from  bombers,  etc.)  This  need  normally  comes  from  a  using  command 
(TAC,  SAC,  MAC,  ATC,  etc.)  and  should  be  stated  in  as  general  terms  as  possible.  This 
heed  is  formalized  via  the  coordination  and  publication  of  a  Statement  of  Operational  Need 
(SON).  After  further  coordination  and  evaluation,  the  Air  Staff  (HQ  USAF)  issues  a  Mission 
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Needs  statement  (MNS)  for  all  SONs  expected  to  lead  to  the  acquisition  of  a  major  system. 
The  MNS  is  "attached"  to  the  Air  Force  Program  Objective  Memorandum  (POM)  which  is 
essentially  nothing  more  than  a  budget  request.  When  the  Secretary  of  Defense  issues  an 
Acquisition  Decision  Memorandum  indicating  his  concurrence  with  the  need,  the  program  is 
bom  (ONCE  FUNDS  ARE  MADE  AVAILABLE). 

With  money,  the  program  enters  the  concept  exploration  phase.  During  this  phase 
Requests  for  Proposals  (RFPs)  are  issued  to  contractors  for  them  to  explore  ways  to  satisfy 
the  particular  need.  The  Systems  Program  Office  (SPO)  and  the  Program  Manager  are 
designated  at  the  beginning  of  the  concept  exploration  phase.  The  Program  Manager  convenes 
a  initial  Test  Planning  Working  Group  (TPWG)  composed  of  all  interested  players.  The  TPWG 
constructs  a  Test  and  Evaluation  Master  Plan  (TEMP)  that  outlines  the  overall  T&E  plan  in 
general  terms.  After  appropriate  data  is  available,  the  Program  Manager  prepares  a  System 
Concept  Paper  (SCP)  that  recommends  a  concept  to  answer  the  need  (i.e.,  kill  tanks  at  night 
with  air-to-ground  missiles).  The  SCP  must  pass  through  the  Defense  System  Acquisition 
Review  Council  (DSARC)  which  will  recommend  approval  or  disapproval  to  the  Secretary  of 
Defense  (OSD).  When  OSD  approves  the  concept.  Milestone  I  has  been  reached.  The  system 
then  enters  the  demonstration  and  validation  phase. 

The  demonstration  and  validation  phase  is  where  fly-offs  occur.  The  TPWG  is 
reconvened,  and  a  revised  TEMP  is  written.  Test  planning,  flying,  and  reporting  are  done  at 
the  local  level  (i.e.,  AFFTC).  From  the  test  results  a  Decision  Coordinating  Paper/Integrated 
Program  Summary  (DCP/IPS)  is  submitted  by  the  Program  Manager  for  DSARC  review  and 
OSD  approval.  When  OSD  picks  a  winner.  Milestone  11  has  been  reached.  The  system  then 
enters  the  full-scale  development  phase. 

Once  again,  a  TPWG  is  called  together  and  a  new  TEMP  is  written.  T&E  is  accom¬ 
plished  and  reports  written  to  provide  data  for  a  production  decision  (Milestone  III).  The 
production  decision  (once  made  at  the  OSD  level),  is  now  made  by  the  applicable  service 
secretary.  After  the  production  decision,  T&E  mostly  consists  of  evaluating  modifications. 

The  above  described  acquisition  process  summarizes  DODD  5000.1  and  DODI  5000.2. 
The  Air  Force  implemented  DOD  5000.1  and  DODI  5000.2  with  AFR  800-2.  DODD  5000.3 
deals  with  T&E  within  this  process.  Specifically,  "Test  and  evaluation  shall  begin  as  early  as 
possible  and  be  conducted  throughout  the  system  acquisition  process  to  assess  and  reduce 
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acquisition  risks  and  to  estimate  the  operational  effectiveness  and  operational  suitability  of  the 
system  being  developed."  The  Air  Force  implemented  DODD  5000.3  with  APR  80-14. 

Testing  to  "...assess  and  reduce  acquisition  risks..."  is  called  Development  Test  and 
Evaluation  (DT&E).  This  testing  is  conducted  by  the  implementing  command,  normally  AFSC. 
Testing  to  "...estimate  the  operational  effectiveness  and  operational  suitability..."  is  called 
Operational  Test  and  Evaluation  (OT&E).  The  Air  Force  Operational  Test  and  Evaluation 
Center  (AFOTEC)  conducts  OT&E  for  all  major  systems  lAW  AFR  980-14.  AFOTEC  is  an 
independent  organization  reporting  directly  to  the  Air  Force  Chief  of  Staff.  Follow-on 
OT&E  for  non-major  systems  is  sometimes  conducted  by  the  using  command.  AFR  80-14 
defines  OT&E  conducted  before  the  production  decision  as  Initial  Operational  Test  and 
Evaluation  (lOT&E)  and  OT&E  conducted  after  the  production  decision  as  Follow-on 
Operational  Test  and  Evaluation  (FOT&E). 

2.3  TESTPIANNING  PROCESS 

A  successful  test  program  begins  with  good  planning.  If  the  plan  is  comprehensive  and 
concise  with  a  clear  understanding  of  questions  to  be  answered,  the  actual  testing  and  report 
writing  will  be  straightforward.  This  means  that  you,  as  a  project  manager,  must  spend  a 
good  deal  of  time  and  effort  planning.  Unfortunately,  you  will  rarely  be  given  adequate  time. 
Therefore,  it  is  essential  that  you  understand  the  test  planning  process  so  you  can  plan 
efficiently  and  effectively. 

Overall  program  management  rests  with  the  SPO.  They  take  the  program  through  the 
system  acquisition  cycle.  Test  data  is  needed  for  the  various  milestone  decisions,  but  the  SPO 
has  no  inherent  capability  to  do  testing.  Therefore,  each  program  has  a  Responsible  Test 
Organization  (RTO)  assigned.  The  RTO  designs  and  conducts  tests  and  reports  back  to  the 
SPO.  Participating  Test  Organizations  (PTOs)  may  be  assigned  to  assist  the  RTO  by  providing 
resources  and  support  that  are  beyond  the  RTO’s  capability.  Support  agencies  assist  each  of 
these  organizations  with  common  tasks.  Specific  RTO  and  PTO  responsibilities  are  described 
in  AFR  80-14/AFSC  Sup  1. 

RTOs  are  appointed  by  AFSC/XT  (with  the  concurrence  of  the  SPO).  Typical  RTOs 
are  the  6510  Test  Wing  (Edwards),  the  3246  Test  Wing  (Eglin),  the  4950  Test  Wing 
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(Wright-Patterson),  or  some  other  test  organizations  based  on  the  nature  of  the  program. 
Normally  at  this  time,  PTOs  are  also  appointed. 

Each  acquisition  program  wUl  contain  some  overall  guidance  for  the  test  and  evaluation 
effort.  Normally,  this  guidance  will  be  in  the  form  of  a  TEMP  that  is  written  by  a  TPWG. 
The  TPWG  is  convened  by  the  Program  Manager  (the  SPO  director)  and  normally  includes 
representatives  from  the  RTO,  PTOs  (if  applicable),  SPO,  contractor(s),  AFLC,  AFOTEC, 
using  command(s),  and  other  involved  agencies. 

The  SPO  formally  requests  test  services  by  sending  a  Program  Introduction  letter  with 
a  Program  Introduction  Document  (PID)  to  the  RTO’s  program  office  (i.e.,  at  Edwards  and 
Eglin  the  office  is  called  "Plans  and  Programs").  The  RTO  program  office  will  ask  the  local 
Project  Manager  (PM)  to  define  this  program  for  them  in  RTO  terms;  Request  for  Test 
Concept.  The  Test  Concept  should  indicate  everything  the  RTO  Project  Manager  needs  to 
satisfy  the  objectives.  If  the  objectives  are  not  crystal  clear,  the  RTO  must  get  them  clarified. 
Based  on  the  Test  Concept,  the  RTO  program  office  will  internally  coordinate  an  effort  to 
list  the  existing  and  needed  resources  and  capabilities  required  to  accomplish  the  test 
objectives.  They  will  conduct  a  mirror  process  and  request  internal  statements  of  capability, 
letters  of  intent  to  provide  services,  etcetera  and  respond  to  the  SPO  (often  referred  to  as 
the  "customer",  but  in  reality,  the  SPO  is  responding  to  their  own  "customer")  with  an 
overall  Statement  of  Capability  (SOC). 

The  SOC  is  nothing  more  than  a  cost  estunate  of  what  the  organization  perceives  as 
needed  and  is  capable  of  doing.  If  the  customer(s)  concurs  with  the  SOC,  they  will  forward 
funds  for  test  planning,  conduct,  and  reporting.  The  product  that  the  customer  is  ultimately 
paying  for  is  a  test  report  (which  normally  includes  a  formal  briefing).  The  relationship 
between  an  RTO  and  PTO  is  virtually  the  same  as  between  a  SPO  and  RTO. 

Normally,  one  person  at  the  RTO  (or  PTO)  is  responsible  for  the  test  planning, 
conduct,  and  reporting.  That  person  is  usually  the  Project  Manager  (PM).  The  next  section 
will  discuss  the  principles  that  this  person  must  apply  in  order  to  effectively  plan  the  test  so 
that  conduct  and  reporting  can  be  accomplished  smoothly  and  efficiently. 


2.7 


2.4  PLANNINGATEST 


DT&E  must  determine  the  degree  to  which  contract  specifications  have  been  met.  But 
that  is  not  enough.  There  are  two  basic  questions  which  must  be  addressed  in  development 
testing: 


(1)  does  the  equipment  meet  specifications  and; 

(2)  does  the  equipment  do  what  it  was  intended  to  do? 

We  may  not  always  be  able  to  obtain  clear  answers  to  the  second  question,  particularly 
in  early  tests  of  complex  systems,  but  we  must  keep  the  question  in  mind. 

The  specifications  in  the  early  design  stages  of  some  systems  may  be  based  on 
experience  and  engineering  judgment.  They  may  not  necessarily  guarantee  that  the  intended 
mission  can  be  accomplished.  If  we  concentrate  on  testing  for  specification  compliance  only 
and  lose  sight  of  mission  suitability,  we  could  find  ourselves  downstream  with  a  system  that 
meets  specification’s  beautifully,  but  will  not  do  the  mission.  In  this  case  we’ve  fallen  into 
Pitfall  Number  1. 

In  a  black  and  white  world,  development  tests  would  be  conducted  under  completely 
controlled  conditions  and  operational  testing  would  be  free-play.  However,  in  the  real  world 
neither  of  these  extremes  is  fully  obtainable.  Remember  that  in  order  to  be  a  valid  test  it 
must  be  a  repeatable  test  and  one  must  control  conditions  to  obtain  repeatable  results.  If  you 
can’t  obtain  repeatable  results,  you  don’t  know  what  results  you  have.  If  tests  indicate  that 
the  system  meets  specifications,  but  you  can’t  get  the  results  in  a  retest,  you  don’t  know 
whether  specifications  were  indeed  met  or  whether  external  influences  produced  the  results. 

How  much  testing  should  you  do?  A  tough  question.  The  operational  aircrew,  your 
ultimate  customer,  may  hold  two  separate  and  conflicting  perceptions  of  you.  While  you’re 
testing  and  retesting,  the  aircrew  in  the  field  (who  desperately  needs  the  system),  thinks  of 
you  as  Nero,  fiddling  while  Rome  bums.  The  same  individual  (seeing  himself  on  the  firing  line 
doing  his  duty  for  God  and  Country),  who  receives  a  system  that  doesn’t  work  as  advertised, 
thinks  of  you  as  another  Washington  bumbler  who  keeps  sending  him  pieces  of  junk.  You  will 
find  yourself  under  conflicting  pressures: 

( 1 )  to  get  the  system  developed  and  deployed; 
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(2)  to  ensure  that  the  system  will  do  its  intended  mission;  and 

(3)  to  maintain  credibility  with  the  user. 

The  adequate  level  of  testing  is  a  judgement  call.  You  must  weigh  the  various  factors 
that  are  involved  in  making  this  determination.  Some  of  the  primary  factors  involved  are  test 
results  to  date,  technological  risk  involved,  and  the  urgency  of  the  requirement.  The  more 
testing  you  do  (not  necessarily  the  more  data  you  generate),  the  more  confidence  you  have 
that  you  will  deploy  a  good  system.  The  less  testing  you  do,  the  greater  the  risk  that  you’ll 
deploy  a  system  that  won’t  work  as  well  as  it  should.  The  happy  medium  is  hard  to  find. 

Testing  is  very  expensive  in  both  time  and  dollars.  Anything  that  is  expensive  can  be 
wasteful  if  not  used  properly.  Program  decisions  depend  heavily  upon  test  RESULTS.  Note 
the  word  RESULTS.  The  number  and  duration  of  tests,  the  test  procedures,  and  the  test 
environment  are  all  important  in  establishing  the  validity  and  confidence  level  of  the  results. 

In  writing  a  test  plan  don’t  get  so  involved  with  the  trees  that  you  lose  sight  of  the 
forest.  The  first  critical  step  in  test  planning  is  to  clearly  identify  the  questions  that  must  be 
answered  and  the  results  that  the  test  should  produce.  The  PID  and  TEMP  should  identify 
critical  issues.  You  must  clearly  identify  these  issues  and  questions  if  you’re  going  to 
structure  a  test  that  will  provide  the  answers.  Otherwise  you’re  groping  through  the  trees 
toward  Pitfall  Number  2;  test  completed,  masses  of  data  in  hand,  and  no  questions  answered. 

If  you  are  getting  the  impression  that  backwards  test  planning  is  recommended,  you 
are  right.  If  you  have  the  questions  and  issues  the  test  must  address  clearly  in  mind,  then  and 
only  then  will  you  be  in  a  position  to  make  realistic  assessments  of  the  data  needed  to  answer 
the  questions  and  resolve  the  issues.  Raw  data  will  frequently  require  data  reduction  and  the 
application  of  analytical  or  statistical  techniques.  We  want  to  produce  results  that  are  clear 
and  meaningful  in  ordinary  language.  We  also  want  to  produce  sufficient  data  to  provide  a 
realistic  confidence  level  that  the  results  reflect  the  true  system  performance.  We  are  dealing 
with  three  interrelated  questions: 

(1 )  how  many  raw  data  points  do  we  need? 

(2)  how  are  we  going  to  reduce  the  raw  data?  and 

(3)  what  analytical  method  will  we  use? 
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Determining  the  answers  to  these  questions  is  an  interactive  process.  Good  answers  to 
these  questions  are  important.  They  help  avoid  Pitfall  Number  2  and  also  permit  efficient  test 
planning.  Ideally,  weTl  test  enough  to  collect  just  the  data  we  need.  In  real  life,  we’ll 
probably  throw  in  a  pad  to  ensure  having  sufficient  data,  while  avoiding  the  expensive 
collection  of  extraneous  data. 

Now,  with  a  handle  on  the  raw  data  we  need,  we  can  take  another  step  backwards  and 
determine  how  we’ll  collect  the  data.  The  nature  and  quantity  of  the  data  should  lead  us 
naturally  to  the  most  reasonable  collection  method.  We  wiU  want  hard  copy  of  some  nature. 
Data  acquisition  methods  vary  in  capacity,  cost,  and  quality  from  high-speed,  multichannel 
magnetic  tape  to  a  stubby  pencil.  Quality,  in  this  context,  means  resolution.  Required  test 
results,  test/detection  equipment,  and  the  recording  medium  should  all  be  consistent.  If  the 
test  results  are  meaningful  only  to  the  nearest  0.5  volt,  it’s  wasteful  to  measure  and  record 
to  a  0.001  volt.  On  the  other  hand,  it  should  be  obvious  that  if  fine  resolution  is  required, 
both  measurement  and  recording  devices  must  provide  it. 

Our  next  backward  step  is  to  determine  data  points  and  recording  times.  We  may 
require  data  from  a  given  point  only  during  portions  of  the  test.  We  may  need  continuous 
data.  In  determining  data  points,  watch  out  for  Pitfall  Number  3:  test/recording  equipment 
or  procedures  contaminating  test  results.  Measuring  the  parameters  of  an  electronic  circuit  is 
fairly  straightforward.  Be  careful  that  you  don’t  become  complacent.  You  can  tap  into  a  data 
collection  point  (both  physically  and  electrically)  and  introduce  unknown  and  undesired 
influences  on  system  performance. 

Stepping  another  pace  to  the  rear,  we  can  now  determine  the  necessary  test  procedures. 
Knowing  the  data  required  and  how  to  collect  it,  we  can  formulate  a  scenario  that  will 
produce  the  necessary  data. 

Procedures  boil  down  to  what,  when,  where,  and  how.  In  developing  step-by-step  test 
procedures,  we  must  also  identify  who.  Some  procedures  must  run  in  sequence;  others  can  be 
run  simultaneously.  People  are  a  major  expense  in  any  test,  so  we  want  to  make  best  use  of 
this  resource.  We  can  keep  the  number  of  people  to  a  minimum  by  careful  planning. 

A  mental  walk-through  (a  deliberate  step-by-step  examination  of  each  sequence  in  the 
test  in  the  test  scenario)  also  enables  us  to  identify,  one  at  a  time,  any  special  test  equipment 
that  we  might  need.  This  mental  walk-through  also  gives  us  a  clear  idea  of  how  the  test 
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item(s),  test  equipment,  and  recording  equipment  all  fit  together.  Most  important,  it  provides 
us  the  information  needed  for  the  final  step  backward  to  site  selection  and  test  setup.  With 
an  image  of  how  everything  fits  and  operates  together,  the  design  of  the  test  setup  is  simple 
and  straightforward.  With  the  entire  test  in  focus,  we  can  identify  site  requirements  including 
physical  parameters  and  required  support  facilities.  Selection  then  becomes  a  simple  matter  of 
identifying  (and  scheduling)  the  most  convenient  site  which  meets  all  requirements. 

The  above  sequence  of  backward  planning  steps  is  summarized  as  follows: 

(1)  Identify  questions  to  be  answered  or  critical  issues  to  be  addressed. 

(2)  Envision  the  test  results  needed  to  answer/resolve  the  questions  and  issues. 

(3)  Determine  the  raw  data  requirements  and  the  data  reduction  and  analytic  methods 
to  be  used  in  producing  the  required  results. 

(4)  Determine  what  resolution  is  required. 

(5)  Identify  data  points,  data  collection,  and  data  capture  methods. 

(6)  Determine  the  test  procedures. 

(7)  Determine  personnel  requirements;  how  many  people  at  what  points  doing  what 

things. 

(8)  Mentally  walk  through  the  test. 

(9)  Design  the  test  setup  and  make  a  site  selection. 

(10)  Write  the  test  plan. 

A  final  note  on  test  planning  --  avoid  Pitfall  Number  4:  undue  optimism.  Undue 
optimism,  particularly  with  respect  to  test  scheduling,  can  create  a  lot  of  problems.  Allow 
sufficient  time.  Anticipate,  particularly  in  the  early  development  cycle,  that  testing  will 
identify  problems.  Problems  normally  mean  delay.  If  you’re  lucky,  the  delay  may  be  short. 

On  the  other  hand,  the  time  to  find  and  fix  a  fault  could  take  several  days  (or  weeks)  which 
would  require  a  suspension  of  the  test.  In  any  case,  the  fix  may  invalidate  portions  of  the 
testing  completed,  which  would  require  some  amount  of  retesting.  Don’t  plan  on  failure,  but 
don’t  schedule  on  a  nothing-can-go-wrong  basis  either.  Remember  Murphy’s  Law. 

One  final  noteworthy  pitfall  has  to  do  not  only  with  the  planning  of  a  test,  but  also 
with  its  conduct  and  reporting.  Pitfall  Number  5:  voluminous  charts,  graphs  and  endless 
tables  of  data  can  be  used  to  clearly  demonstrate  the  authenticity  of  a  bunch  of  nonsense. 
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TEST  MANAGEMENT  PROJECTS 

Your  test  management  project  is  the  "graduation  exercise."  The  key  word  is  manage¬ 
ment.  Within  the  constraints  of  school  capabilities,  the  project  requirements  for  planning, 
documentation,  conduct,  and  reporting  will  closely  mirror  the  real  world.  Since  AFFTC  is  the 
closest  world,  its  procedures  will  dominate.  They  are,  however,  typical  of  other  centers. 

The  project  begins  when  the  TPS  Test  Management  Branch  (USAFTPS/STT)  receives  a 
Program  Introduction  Document  (PID)  from  the  customer  through  the  staff  project  monitor. 
The  staff  project  moniter  is  used  at  TPS  to  fill  in  for  positions  not  covered  and  to  assist  the 
test  team  in  managing  their  project.  In  real  world  test  units  there  will  not  be  a  staff  moniter 
to  assist,  but  there  probably  will  be  a  consultant  of  some  type.  A  Program  Introduction 
letter  accompanying  the  PID  will  be  forwarded  to  the  student  Project  Manager  (PM)  along  with 
the  Request  For  Test  Concept.  The  PID  should  contain  objectives  for  your  test.  If  it 
doesn’t,  contact  your  staff  moniter  immediately,  as  you  need  to  know  where  you  are  going  if 
you  expect  to  make  progress  towards  it.  You  will  need  to  refine  the  objectives. 

Next  the  PM  and  student  test  team  prepare  a  Test  Concept.  It  is  prepared  in  accor¬ 
dance  with  appendix  A  of  this  text  and  AFFTCR  27-4.  It  consists  of  a  cover  letter  with  two 
attachments.  The  first  attachment  is  an  AFSC  Form  5341  and  the  second  attachment  is  a 
milestone  chart  (AFSC  Form  103,  which  may  be  modified  as  necessary).  The  Test  Concept  is 
forwarded  to  USAFTPS  Program  Analysist  in  the  Resources  Branch  (USAFTPS/CCR)  and 
serves  to  identify  resources  required  to  obtain  the  test  results  required  to  answer  the  critical 
questions  identified  in  the  PID.  The  Test  Management  Branch  (USAFTPS/STT)  may  ask  you 
to  route  your  Test  Concept  through  your  Staff  Moniter  and/or  STT.  For  major  projects, 
AFFTC  Program  Office  may  serve  in  this  capacity.  The  Program  Analysist  wUl  solicit 
Statements  of  Capability  from  each  agency  you  have  requested  support  from  in  your  Test 
Concept,  and  coordinate  the  development  of  a  Project  Directive.  You  should  have  contacted 
each  of  these  support  agencies  already.  USAFTPS/CCR  determines  the  cost  (in  dollars  and 
cents)  of  the  required  support.  If  the  cost  is  within  the  school’s  (and/or  customer’s)  budget, 
the  concept  will  be  approved.  Otherwise,  the  concept  wUl  have  to  be  revised  and  the  Project 
Directive  wUl  be  delayed. 


By  now,  a  project  folder  should  have  been  developed.  Appendix  B  contains  cover  pages 
for  each  of  the  six  sections.  Use  these  or  make  copies  then  all  you  have  to  do  is  fill  up  the 
folder  as  you  collect  documents.  The  project  folder  is  very  valuable  so  don’t  lose  it. 

The  student  PM  then  schedules  a  TPWG  meeting.  The  PM  should  invite  representatives 
from  all  agencies  from  which  he/she  anticipates  support,  plus  any  other  individuals  he/she 
deems  necessary.  Before  your  TPWG,  you  should  have  available  a  fairly  complete  test  plan 
draft.  The  experts  are  at  the  TPWG  to  assist  in  finalizing  your  plan,  not  to  write  it  for  you. 

The  next  step  is  to  write  the  final  draft  of  the  test  plan.  It  should  be  easy.  At  this 
point,  you  already  know  your  specific  test  objectives,  how  you  plan  to  conduct  the  test, 
what  support  is  needed,  and  who  will  provide  it.  You  could  not  have  prepared  a  realistic  Test 
Concept  without  this  information.  It  now  becomes  a  matter  of  writing  it  all  down  in  the 
proper  format.  The  TPS  format  is  contained  in  Appendix  C.  It  closely  matches  the  AFFTC 
format.  Remember,  a  test  plan  should  be  comprehensive  enough  to  allow  flight  test 
personnel,  previously  unfamiliar  with  the  test,  to  conduct  the  test  and  safely  satisfy  all 
objectives.  You  need  to  be  familiar  with  the  following  references: 

a.  USAFTPS  Operating  Instruction  (OI)  51-10,  Curriculum  Flight  Test  Plans.  This 
regulation  specifies  the  general  content  of  student  test  plans  and  specifies  review  and  approval 
actions.  In  general,  if  your  test  program  is  judged  to  be  low  risk,  the  approval  authority 
rests  with  the  TPS  Commandant.  Otherwise,  the  plan  must  be  approved  by  the  AFFTC/CC, 
after  approval  by  the  USAFTPS/CC  and  6510  TESTW/CC. 

b.  AFFTCR  127-3,  Safety  Planning  for  AFFTC  Tests.  Every  test  plan  must  include 
and  AFSC  Form  5028  (and  AFSC  Form  5028a  if  required)  as  the  last  appendix. 

c.  AFFTCR  80-1 ,  Test  Plans.  Every  test  plan  must  include  a  comment  concerning 
OPSEC  and  COMSEC.  These  statements  should  be  among  the  items  included  in  the  Introduc¬ 
tion  Section. 

d.  AFSCP  127-2,  Flight  Safety  Planning  Guide  for  Flight  Testing. 

Next,  you  will  present  your  plan  to  a  combined  technical  and  safety  review  board  to 
ensure  that  adequate  planning  and  preparation  have  been  accomplished  prior  to  submitting  the 
plan  to  USAFTPS/CC  (or  AFFTC/CC,  if  required)  for  final  approval. 

The  technical  review  board  (TRB)  starts  with  a  briefing  from  the  student  test  team. 

The  briefing  should  include  a  summary  of  general  and  specific  test  objectives,  rationale  for  the 
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plan,  and  proposed  test  conduct.  USAFTPS  01  51-10  contains  valuable  information  on  the 
conduct  of  the  SRB  and  should  be  consulted.  Following  the  briefing,  an  informal  discussion  / 
question  and  answer  period  is  conducted  addressing  the  test  objectives,  status  of  preparations 
and  planning,  predicted  test  results,  action  taken  to  minimize  risks,  emergency  procedures, 
go  and  no-go  criteria,  alternative  courses  of  action,  and  other  items  important  to  test 
planning.  Obviously,  the  test  team  that  has  already  thought  through  these  questions  and 
included  the  answers  in  the  test  plan  and  briefing  will  be  better  prepared  for  the  technical 
review.  The  test  plan  may  be  approved  as  presented,  or  the  TRB  president  may  dictate 
modifications.  After  making  any  required  modifications,  the  test  plan  is  submitted  for  final 
coordination  and  approval.  To  expedite  approval,  you  should  outline  required  modifications 
on  the  coordination  page. 

The  safety  review  board  (SRB)  will  normally  start  immediately  following  the  TRB. 

The  board  is  chaired  by  a  member  of  the  AFFTC  safety  office  (AFFTC/SE).  The  purpose  of 
the  safety  review  is  to  identify  hazards  generated  by  the  test  and  to  ensure  that  adequate 
action  is  taken  to  eliminate  or  control  these  hazards  to  an  acceptable  level  of  risk.  The  board 
will  also  advise  the  Commandant  and  other  supervisors  of  the  degree  of  risk  that  the  planned 
test  will  present.  MIL  STD  822A  defines  a  mishap  as  an  unplanned  event  or  series  of  events 
that  result  in  death,  injury,  occupational  illness,  or  damage  to  or  loss  of  equipment  or 
property.  Loss  or  damage  to  equipment  that  results  from  intended  functioning  is  not 
considered.  A  hazard  is  defined  as  an  existing  or  potential  condition  that  can  result  in  a 
mishap.  The  hazard  may  be  a  result  of  personnel  error,  environment,  design  characteristics, 
procedural  deficiencies,  or  system  or  component  failure  or  malfunction.  MIL  STD  822A 
provides  the  following  definitions  as  a  qualitative  measure  of  hazards. 

Category  I  (Catastrophic)  -  May  cause  death  or  system  loss. 

Category  II  (Critical)  -  May  cause  severe  injury,  severe  occupational  illness,  or  major 
system  damage;  or  requires  immediate  action  to  prevent  a  Category  I  hazard. 

Category  III  (Marginal)  -  May  cause  minor  injury,  minor  occupational  illness,  or  minor 
system  damage;  or  requires  action  to  prevent  a  Category  II  hazard. 

Category  IV  (Negligible)  -  Will  not  result  in  injury,  occupational  illness  or  system 
damage. 


2.14 


The  PM  is  responsible  for  ensuring  that  the  AFSC  Form  5028  (and  5028a  if  necessary) 
is  prepared  for  presentation  to  the  safety  review  board.  The  SRB  will  discuss  the  hazards  and 
minimizing  procedures  that  are  on  the  AFSC  Form  5028a,  Test  Hazard  Analysis  (THA).  The 
board  can  make  changes,  deletions  or  additions  to  the  THA.  A  review  of  AFFTCR  127-3  and 
USAFTPS  OI  51-10  will  save  you  a  lot  of  hassle  at  the  board,  and  prevent  a  lot  of  repeat 
paperwork.  Once  the  SRB  has  made  changes  (if  any)  to  the  THA,  an  overall  risk  assessment 
will  be  made.  Then  the  board  will  adjourn.  During  the  SRB  it  is  important  that  someone 
from  the  test  team  be  appointed  to  take  notes.  After  any  required  changes  are  made  and  the 
remarks  section  is  filled  in,  all  members  of  the  SRB  must  sign  the  Form  5028.  You  are 
cleared  to  begin  flight  testing  when  US  AFTPS/CC  (or  AFFTC/CC  )  signs  the  Form  5028 
(assuming  the  test  plan  is  signed).  Since  any  changes  to  the  test  plan  require  a  repeat  of  this 
entire  process,  it  is  worth  your  while  to  do  it  right  the  first  time. 

A  statement  concerning  scheduling  is  now  warranted.  Just  because  you  don’t  have  a 
signed  test  plan  and/or  Form  5028  doesn’t  mean  you  can’t  SCHEDULE  flights.  AFFTC  (as 
most  other  centers)  requires  lead  time.  You  need  to  have  your  requests  for  airplanes  and  any 
other  resources  into  USAFTPS  scheduling  at  least  TWO  WEEKS  in  advance.  Always  remember 
that  it  is  very  easy  to  cancel  at  the  last  minute,  but  almost  impossible  to  add-on  at  the  last 
minute.  Keep  in  mind  that  Senior  Management’s  ultimate  STOP  to  prevent  flying  a  test  they 
haven’t  approved  is  the  flying  schedule,  so  it  is  in  your  best  interests  to  be  open  and  up-front 
about  your  scheduling  needs  and  your  intentions.  Your  project  start  time  will  be  fixed  by  the 
curriculum  integrated  schedule.  Do  not  wait  until  after  your  TRB/SRB  to  schedule  missions. 

During  the  active  testing  phase  of  any  project,  briefings  will  have  to  be  given  in  order 
to  "keep  the  boss  informed. "  At  AFFTC  these  are  called  Program  Assessment  Reviews 
(PARs).  You  will  be  required  to  brief  the  status  of  your  project  to  the  Commandant  at 
approximately  the  25%  and  75%  completion  points  of  the  active  test  phase.  These  briefings 
are  normally  held  at  the  weekly  ops  meetings.  Appendix  D  contains  detailed  instructions.  The 
slide  formats  in  Appendix  D  are  intended  for  reproduction. 

Also,  during  the  active  testing  phase  of  any  project  you  will  have  to  "keep  the 
customer  satisfied."  Your  staff  project  monitor  will  play  the  role  of  SPO  director.  Keep 
your  staff  project  monitor  informed  throughout  the  course  of  the  project,  but  it  is  especially 
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critical  during  the  periods  leading  up  to  the  TPWG  and  after  active  testing  but  before  the  final 
report. 

The  final  products  of  your  efforts  will  be  an  oral  and  a  written  report.  The  report 
formats  will  be  specified  in  the  Phase  Planning  Guide.  The  written  report  format  may  vary, 
depending  upon  the  needs  of  the  test  requestor  ("customer").  The  intent  of  the  oral  report  is 
to  simulate  briefing  a  general  officer  on  the  RESULTS  of  your  test  as  concisely  and  clearly  as 
possible. 


2.6  TESTMANAGEMENTPROJECTSUMM^ 

Outlined  below  is  a  one  page  summary  (checklist)  of  what  happens  and  what  you  need 
to  accomplish  for  your  test  management  project. 

1 .  Varied  and  extensive  coordination  will  occur  to  identify  a  project  for  the  Test 
Management  Phase. 

2.  A  TPS  staff  moniter  will  be  assigned  by  Director  of  Student  Training. 

3.  You  should  recieve  a  PI,  PID,  and  Request  for  Test  Concept. 

4.  You  will  formalize  objectives  and  prepare  a  Test  Concept  with  attachments. 

5.  You  will  contact  support  agencies  and  determine  their  capabilities  related  to  your 
test. 

6.  You  will  coordinate  Test  Concept  package  and  forward  to  Program  Analysist. 

7.  You  will  start  test  plan  draft. 

8.  PA  should  solicit  Statements  of  Capability  based  on  your  Test  Concept  and 
prepare  project  directive. 

9.  You  will  schedule  a  TPWG  and  invite  experts. 

10.  You  should  recieve  a  Project  Directive  authorizing  you  to  expend  funds  against 
your  project. 

11.  You  will  complete  your  test  plan  draft  and  distribute  copies  to  TPWG  members. 

12.  You  will  conduct  your  TPWG. 

13.  You  will  revise  your  test  plan  draft,  submit  it  for  grading,  and  distribute  copies 
to  TRB/SRB  members. 

14.  You  will  conduct  a  TRB  and  SRB. 

15.  You  will  make  required  corrections  to  your  test  plan  and  coordinate  the  test  plan 
for  test  plan  approval. 

16.  You  will  prepare  a  Safety  Review  approval  package  and  coordinate  it  for  approval. 

17.  After  your  test  plan  and  safety  package  are  approved  you  may  conduct  your  tests. 

18.  You  wiU  conduct  two  PAR  briefings  for  the  Commandant. 

19.  You  will  coordinate  any  test  plan  and  safety  paperwork  changes  LAW  existing 
procedures. 

20.  After  you  have  finished  testing,  you  will  close  out  your  safety  paperwork. 

21.  You  will  schedule  and  attend  a  report  pre-writers  meeting  with  6510  TW/DOE. 

22.  You  will  prepare  a  draft  final  written  report. 

23 .  You  will  schedule  a  Technical  Coordination  Meeting  (TCM)  with  6510  TW/DOE 
and  distribute  draft  copies  of  your  report  to  participants  one  week  prior. 

24.  You  will  conduct  a  formal  oral  presentation  of  your  results. 

25.  You  will  hold  a  TCM  to  assist  in  preparation  of  your  final  written  report. 

26.  You  will  submit  a  final  written  report  that  you  are  proud  of. 

27.  You  will  say  "Whew!,  I’m  glad  that’s  over." 


GOOD  LUCK!!!!  ENJOY  YOUR  PROJECT. 
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APPENDIX  A 

TEST  CONCEPT 

INSTRUCnONS  AND  SAMPLES 
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USAF  TEST  PILOT  SCHOOL 


TEST  CONCEPT 


For  a  given  test  project,  you,  as  project  manager,  will  receive  a  Program  Introduction 
letter.  Attached  to  it  will  be  a  Request  For  Test  Concept  and  Program  Introduction 
Document.  Your  tasking  will  be  to  prepare  a  Test  Concept  Letter  with  appropriate  attach¬ 
ments.  The  attachments  are: 

1 .  TEST  CONCEPT/TEST  RESOURCE  REQUIREMENTS  -  AFSC  Form  5341 

2.  Notes  to  AFSC  Form  5341 

3 .  PROGRAM  SCHEDULE  -  AFSC  Form  103  (May  be  modified) 

After  you  submit  your  Test  Concept  to  your  Program  Analysist,  you  should  receive 
Statements  Of  Capability  (commitments  in  writing)  from  each  organization  you  requested 
support  for  in  your  Test  Concept.  After  all  Statements  of  Capability  have  been  reviewed. 
Management  will  issue  a  Project  Directive  authorizing  support  for  your  project.  This  Project 
Directive  is  your  authority  to  expend  funds  against  your  project  (start  your  test).  However, 
this  is  not  authorization  to  start  flying. 

NOTE:  A  blank  modified  AFSC  Form  103  is  included  to  simplify  your  work! 
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DEPARTMENT  OF  THE  AIR  FORCE 

USAF  TEST  PILOT  SCHOOL 
EDWARDS  AIR  FORCE  BASE.  CALIFORNIA  93523 


REPLY  TO 

ATTN  OF;  LiSAFTPS/  ( Cap  t  Shelley?  7‘'3691)  2  Sep  87 

SUBJECT:  pp_4c/p-i^  Agility  (Metrics  Test  Concept 

USAFTPS/  CCR 

1.  The  test  concept  for  the  USAFTPS  Class  87A  RF-4C/F-16 
Agility  lietrics  Evaluation  is  attached. 

2.  The  general  objective  of  this  evaluation  is  to  identify 
flight  test  maneuvers,  data  measurement  and  analysis 
techniques,  and  data  presentation  formats  to  quantify 
aircraft  agility. 

3.  The  requesting  organization  is  USAFTPS/  STX  Major  Steven 
J.  Pitotti,  USAF.  is  the  user  project  officer-  The  RF-AC/ 
F-’16  test  team  is  the  responsible  test  organization  for  this 
limited  test  program. 

4-  The  following  members  of  USAFTPS  Class  87A  have  been 
assigned  to  the  test  team: 

a.  Test  Team  Director  -  Captain  Shelley  (USAF) 

b.  Project  Safety  Officer  -  Captain  Jimenez  (USAF) 

c.  Project  Pilot  ~  Captain  Cantoni  (lAF) 

d.  Project  Engineer  -  Captain  Cate  (USAF) 

e.  Project  UlSO  -Captain  Buechter  (USAF) 

f.  Project  Engineer  -Captain  Zanatta  (lAF) 

5.  A  total  of  20  RF-4C  and  10  F- 1 6  sorties  are  p 1 anned - 
14  RF-4C  sorties  will  initially  be  used  to  develop  and 
validate  flight  test  and  data  reduction  techniques.  The 
remaining  six  RF-4C  sorties  will  be  used  to  qualitatively 
verify  previous  results  and  to  assess  the  correlation  of 
agility  metrics  with  pilot  ratings  of  closed  loop  operational 
handling  tasks.  The  F— 16  sorties  will  be  flown  last,  and 
will  be  used  to  demonstrate  and  validate  the  universality  of 
the  results- 


RAN  E.  SHELLEY,  Capt,  USAF  2  Atch 

Project  Manager 

1.  AFCS  Form  5341 

2.  AFSC  Form  103 
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TEST  CONCEPT/TEST  RESOURCE  REQUIREMENTS 
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AFSC  S3  4>  CONTINUED  ON  NEXT  PAGE 


TEST  CONCEPT/TEST  RESOURCE  REQUIREMENTS 


AFSC  S3Hf 

Notes  for  AFPTC  Form  : 

1.  RF-4C  flights  will  be  used  for  FTT  validation  and  primary 
data  acquisition.  Telemetry  aircraft  required  (Tail  #  850,  744, 
941).  SI  preflight  required. 

2.  Two  F-4  sorties  (CF-4/CF-5)  are  needed  to  complete  aircraft 
checkout  for  one  of  the  project  pilots.  Three  F-4  sorties  will 
be  flown  with  an  IP  to  certify  test  team  pilots  for  FTT's. 

3.  T-38  sorties  will  be  used  for  targets  during  closed  loop 
tracking  tasks  for  four  of  the  F-16  sorties  and  six  of  the  RF-4 
sorties . 

4.  F-16B  sorties  will  be  used  for  FTT  verification  and  data 
collection  in  an  airplane  other  than  the  primary  test  aircraft. 
F-16  must  be  instrumented  for  quantitative  results.  Must  have 
access  to  F-16  instrumentation  data  reduction  facilities. 

5.  SPORT/DARC  needed  for  traffic  advisories  and  deconf liction . 

6.  Discrete  frequencies  needed  for  a  2  hour  block  during  TM 
operations . 

7.  Two  Genisco  magnetic  tapes  needed  to  replace  two  cassettes 
that  are  broken. 

8.  Six  one  hour  simulation  periods  in  the  NASA  F-18  simulator  to 
practice  FTT*s.  Point  of  contact:  Maj  Harry  Walker,  TE^R, 
7-3457. 

9.  SPORT  positioning  information  may  be  required  for  incremental 
pitch  angle  FTT. 

10.  TPS  telemetry  room  required  for  all  RF-4  test  sorties. 

11.  F-4  IP  required  for  aircrew  check  out  and  clearing  test  team 
pilots  to  fly  new  FTT's- 

12.  T-38  IP  required  for  formation  flight  lead. 

13.  F-16  IP  required  for  F-16  sorties. 
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AFSC  1QO  PREVIOUS  EDITIONS  ARE  OBSOLETE, 

Overprinted  for  USAFTPS  training  purposes 


APPENDIX  B 

PROJECTFOLDER 

INSTRUCTIONS  AND  COVER  PAGES 
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US  AF  TEST  PILOT  SCHOOL 


PROJECT  FOLDER 


You  will  be  required  to  maintain  a  project  folder  which  must  be  available  to  any  staff 
supervisor  at  all  times.  Since  these  may  be  real  world  projects  and,  in  all  cases,  are  projects  of 
interest  to  supervisors  outside  the  School,  questions  get  asked  and  must  be  answered.  Use  a 
brown,  six-divider  folder.  Cover  pages  for  each  of  the  six  sections  are  included  here. 

Your  project  folder  is  not  to  be  confused  with  a  Safety  Review  approval  package. 

These  are  two  seperate  items.  Do  not  send  your  project  folder  with  your  safety  paperwork 
for  senior  management  approval,  as  you  will  probably  never  see  your  other  project  material 
again.  See  the  Unit  Test  Safety  Officer  for  more  guidance. 
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BACKGROUND 


SUPPORTING  DOCUMENTS 
RECEIPTS 
NOTES 
TELECONS 

MEMOS  FOR  RECORD 
MEETING  ATTENDENCE  /  MINUTES 
CORRESPONDENCE 
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MANAGEMENT  INFORMATION 


PROJECT  DIRECTIVE 
STATEMENTS  OF  CAPABILITY 
TEST  CONCEPT  (LTR  WITH  AFSC  Forms  5341  &  103) 
PROGRAM  INTRODUCnON  LETTER 
REQUEST  FOR  TEST  CONCEPT 
PROGRAM  INTRODUCTION  DOCUMENT 
PROJECT  ASSIGNMENT  LETTER 


SAFETY  REVIEW 


AFSC  FORMS  5028b 
AFSC  FORMS  5028a 
AFSC  FORM  5028 

SAFETY  REVIEW  STAFF  SUMMARY  SHEET 
SRB  MEETING  MINUTES  /  ATTENDENCE 
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TEST  PLANNING 


TEST  PLAN  AMENDMENTS 
TEST  CARDS 
TEST  PLAN 

AFSC  FORMS  5232b  (Drafts,  final,  and  amendments) 
TRB  MEETING  ATTENDENCE  /  MINUTES 
TPWG  MEETING  ATTENDENCE  /  MINUTES 
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BRIEFING  SLIDES 


FINAL  ORAL  PRESENTATION 
PAR 
SRB 
TRB 
TPWG 
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FIN  AT.  REPORT 


Appendix  C 


Expanded  USAFTPS  Test  Plan  Format 
(Instructional  Version) 
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AFFTC  TEST  INFORMATION  SHEET  (TiS) 

(  Qual  test  PROGRAM) 


F-15E  Qualitative  Evaluation 


TlS  TYPE  LOCATION  OF  TEST 

12  PLAN  □  PROCEDURAL  Luke  AFB 


DATE 

29  Jan  92 

PAGE  1  OF  PAGES 

VEHICLE  TYPE 

F~15E 

TIS  NUMBER 

EFFECTIVITY 

N/A 

REVISION 

Basic 

TESTING  ACTIVITY 
U SAP TPS  /  STB 


HAZARDOUS/UNUSUAL  TEST 
No  /  No _ 


This  USAFTPS  curriculum  test  plan  was  reviewed  in  accordance  with  the  procedures  outlined  in  USAFTPS 
OI  51-10  and  not  specifically  in  accordance  with  the  instructions  contained  in  AFFTCR  80-1. 


( This  page  is  a  sample  of  AFSC  Form  5232b,  reduced  size.  The  above  statement 
is  mandatory  for  TPS  curriculum  test  plans.  The  form’s  actual  uses  vary  from 
unit  to  unit  and  base  to  base,  but  for  detailed  test  plans  it  serves  as  an  overall 
cover  and  coordination  page.  Think  of  it  as  a  Staff  Summary  Sheet.  "The  form 
should  contain  background  and  authority  information  and  should  lead  the  reader 
to  a  better  understanding  of  the  test."  Continue  on  the  back  side  if  necessary.  If 
you  have  TRB  directed  changes  to  your  test  plan,  please  outline  them  here,  for 
the  reviewers’  and  approvers’  ease.  If  you  amend  your  test  plan,  use  another  of 
this  same  form,  annotate  the  "REVISION"  block  above  accordingly,  and  write 
out  the  changes.  In  this  case,  you  only  need  signatures  of  the  affected  reviewers 
and  the  approver.  If  the  form  has  insufficient  signature  blocks  for  all  your 
reviewers,  make  additional  signature  blocks  above  the  pre-printed  ones.  Please  use 
some  discretion.  For  Qual  test  plans,  overtype  the  bottom  REVIEW  block  as 
APPROVE  for  the  second  approver.) 


PREPARING/CDORDINATING  OFFICIALS 


action 

OFFICE  SYMBOL 

SIGNATURE 

— 

PREPARE 

STB 

COMMENTS 
CONCUR  added 

YES  I  NO  YES  \  NO 


REVIEW 

STT 

APPROVE 

ST 

DO 

L_ 

AFSC  Form  5232b,  MAR  88 


PREVIOUS  EDITION  WILL  BE  USED 
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(Sample) 


TABLE  OF  CONTENTS 


SECTION 


PAGE 


LIST  OF  ABBREVIATIONS .  3 

LIST  OF  FIGURES . 

LIST  OF  TABLES . 

1.0  GENERAL . 

2.0  OBJECTIVES . 

3.0  ORGANIZATION . 

4.0  RESPONSIBILITIES  AND  SPECIAL  SUPPORT . 

5.0  SCHEDULE . 

6.0  TEST  ITEM  DESCRIPTION . 

7.0  INSTRUMENTATION . 

8.0  TEST  CONDITIONS,  PROCEDURES,  METHODS,  AND  TECHNIQUES  . . . 

9.0  DATA  PROCESSING,  REDUCTION,  AND  ANALYSIS . 

REFERENCES . 

APPENDIX  A  (TITLE) .  A-1 

A-1.0  (TITLE) .  A-2 

APPENDIX  X  TEST  SAFETY  REVIEW  AND  DOCUMENTATION .  X-1 

X-1.0  SAFETY  CONSIDERATIONS  AND  PLANNING .  X-2 

X-2.0  TEST  PROJECT  SAFETY  REVIEW  (AFSC  Form  5028) .... 

X-3.0  TEST  HAZARD  ANALYSIS  (AFSC  Forms  5028a) . 

( NOTE:  Safety  paperwork  is  always  last  appendix) 
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USAFTPS  TEST  PLAN  FORMAT  -  Start  to  Finish! 


REMINDER:  "Test  plans  contain  sufficient  information  to  directly  develop  flight  test  cards  and 
for  management  to  discern  the  overall  technical  approach  being  taken.  Therefore,  test  plans  are 
the  key  documents  that  describe  the  specific  tests  to  be  accomplished  and  how  they  will  be 
accomplished."  AFFTCR  80-1 


AFSC  Form  5232b  ( Used  somewhat  like  a  Staff  Summary  Sheet;  to  explain  the  gist  of  the  Test 

Plan  and  for  coordination.  The  title  should  be  a  thorough  representation  of 
the  entire  test  effort.  Once  a  TIS  #  is  assigned  the  plan  is  a  fixed  document 
and  subsequent  changes  are  numbered  as  revisions.  It  is  common  to  go  to  a 
TRB  with  REVISION  Three  in  hand.  ) 


TABLE  OF  CONTENTS 

( see  attached ) 

UST  OF  ABBREVIATIONS 

( These  three  lists  may  be  on  same  page,  if  short ) 

USTOFHGURES 

ft 

LIST  OF  TABLES 

If 

1.0  GENERAL 

1.1 

INTRODUCnON: 

(  Who,  what,  when,  where,  and  why  ) 

1.1 

BACKGROUND: 

( What  makes  this  test  worth  doing?  Mention  critical 
technical  issues  here  or  refer  to  an  appendix. ) 

1.3 

RELATED  TESTS:  ( Past,  present,  and  future ) 

1.4 

SCOPE: 

( Simple  test  or  are  many  organizations  involved?  What  will  it  take  to  get  the 
needed  data?  Mention  maximum  number  of  sorties,  if  appropriate ) 

1.5 

LOCATION(S): 

1.6 

CRITICAL  DATES  AND  SUSPENSES:  ( What  are  your  external  time  constraints?  What 

does  the  boss  need  advance  warning  for? ) 

1.7 

AUTHORITY: 

(  Usually  Commandant;  Also  reference  JON,  PID,  and  priority  ) 

1.8 

SAFETY: 

(  Short  discussion;  What  rules;  Refer  reader  to  last  appendix ) 
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1.9  SECURITY: 


( Look  at  required  regulations  and  reference  them.  Partially 
classified  tests  need  detailed  attention  here. ) 


1.10  REPORTING: 

2.0  OBJECTIVES 

2.1  GENERAL: 

2.2  SPECIHC: 

2.2.1  - 


( To  whom?  Will  it  answer  objectives?  ) 


(  Qearly  defined?  "Success"  criteria  outlined  for  each?  ) 

( Overall  objective  for  whole  program ) 

( As  separate  and  unique  as  possible;  Quantifiable  and 
answerable;  Put  some  effort  here  ) 


3 . 0  ORGANIZATION  ( Layout  for  this  section  will  vary  but  should  cover  the  listed  items  in 

a  hierarchical  fashion.  Including  a  wiring  diagram  showing  who  is 
making  inputs  to  whom  is  often  helpful. ) 

3 . 1  The  Requesting  Agency  is .  ( Customer ) 

3.2  The  Program  Office  is . 

3.3  The  Responsible  Test  Organization  is .  ( You ) 

3.3.1  The  Project  Manager  is . 

3.3.2  The  Project  Safety  Officer  is . 

3.3.3  - 

3 .4  The  Participating  Test  Organizations  are . 

3.5  The  Support  Agencies  are . 

3.6  - 


(WIRING  DIAGRAM  ??) 
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4.0  RESPONSronjmES  AND  SPECIAL  SUPPORT  ( This  section  is,  in  some  ways,  like  a  contract. 

Get  commitments  from  outside  agencies  before 
listing  their  responsibilities  here. ) 

4 . 1  The  Project  Manager  wiU: 

4.1.1  - 

4.x  The  Project  Safety  Officer  will: 

4.X.1  - 

4.x  The  Participating  Test  Organization  will: 

4.x  - -  Support  aircraft,  Simulation,  Range,  Facilities,  Equipment, 

Photos,  Special  data  handling,  etc.  are  reasonable  subjects  for 
the  Responsibilities  and  Special  Support  section. 


5.0  SCHEDULE  (Tables  and/or  diagrams  are  helpfiil) 

5.1  GENERAL 

5.x  -  Phases,  Training,  Critical  Sorties,  Priorities,  Built-in  Stops, 

Decision  Points,  etc.  are  reasonable  subheadings  for  the 
Schedule  section. 

(MILESTONE  CHART  ??) 


6.0  TEST  ITEM  DESCRIPTION  ( Refer  detailed  descriptions  to  an  appendix ) 

6. 1  PRIMARY  TEST  ITEM:  Production  relevance.  Software  version 

6.x  Modifications  needed  to  anything.  Compromises,  Configuration  Control 
6.x  Supporting  items,  aircraft,  or  systems 
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7.0  nsrSTTlUMENTATrON 


( Refer  detailed  parameter  lists  to  an  appendix ) 


7 . 1  GENERAL:  Description,  Type,  Accuracy,  Sensitivity,  Calibration,  Go-no  go 

( Are  instnimentation  requirements  different  for  each  specific  objective? ) 

7.2  AIRBORNE: 

7.3  GROUND: 

8.0  TEST  CONDITIONS,  PROCEDURES,  METHODS,  AND  TECHNIQUES 

(This  is  the  heart  of  the  test  plan  and  should  directiy  relate  to  the  objectives.  Write  out  as  much  as  possible,  please!) 


8.1  GENERAL: 

8.1.1 

( However  you  need  to  start  this  section ) 

8.1.x 

TRAINING: 

(  This  is  your  responsibility!  Do  some! ) 

8.1.x 

SIMULATION: 

( Eveiything  can  and  should  be  simulated  to  some  degree  ) 

8.1.x 

LIMITATIONS: 

( Also  mention  risk  of  exceeding.  Safety  hazard  minimizing 
procedurees  belong  in  Safety  Appendix ) 

8.1.x 

TEST  ENVELOPE: 

( A  picture  of  test  and  aircraft  envelopes  is  required ) 

8.1.x 

TEST  POINTS: 

( Long  lists  belong  in  an  appendix;  need  to  specify  altitudes, 
airspeeds,  g  loads,  configuration  ,  etc. ) 

8.1.x 

Build-up  approach.  Priorities,  Repeats  vs.  statistical  significance. 

Test  point  abort  procedures 

8.1.x 

Process  of  review  before  proceeding  with  subsequent  sorties 

8.1.x 

Alternate  missions.  Back-up  options,  Weather  criteria 
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8.2  SPEanC: 


8.2.1 


Spell  these  out  step-by-step:  Conditions,  Data  band,  Proce¬ 
dures,  Methods,  Techniques,  Data  collection.  Repeatability, 
In-flight  clearance,  etc. 

( Consider  matching  to  each  specific  objective ) 


9.0  DATA  PROCESSING,  REDUCHON,  AND  ANALYSIS 

9.1  GENERAL:  Hand,  Recorded,  Computer,  Format,  Data  review  and  turn 

around.  Real  time.  Data  tolerance.  Data  management 

9.2  SPECIFIC:  (Consider  matching  to  each  specific  objective) 

AFSCP  127-2,  AFFTCR  127-3,  USAFTPS  01  51-10,  plus 
many  others;  Be  thorough! 

(  Titled  to  be  informative  ) 

TEST  SAFETY  REVIEW  AND  DOCUMENTATION  ( Always  last  append. ) 

SAFETY  CONSIDERATIONS  AND  PLANNING  ACCOMPUSHED 
TEST  PROJECT  SAFETY  REVIEW  (AFSC  Form  5028) 

TEST  HAZARD  ANALYSIS  (AFSC  Forms  5028a) 


REFERENCES 

APPENDIX  A 
APPENDIX  X 
X-1.0 
X-2.0 
X-3.0 


X-3.1 


NOTE:  ^  GENERAL  section  (1.1  -  1.10),  OBJECTIVES  section  (2.1  &  2.2), 
sections  3.0,  4.0,  5.0,  6.0,  6.1,  7.0,  8.0,  8.1,  8.2,  a  depiction  of  the  test 
envelope,  and  section  9.0  are  always  required.  For  other  than  a  Qual  test  plan 
lAW  TPS  OI  51-9,  REFERENCES  and  Safety  Paperwork  (Appendix  X)  are  always 
required.  TABLE  OF  CONTENTS  is  only  required  if  it  aids  reviewing  the  test 
plan.  Remaining  subheadings  and  other  items  are  flexible  depending  on  your  test, 
but  the  material  should  be  present  in  some  form. 
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APPENDIX  D 

PROGRAM  ASSESSMENT  REVIEW 

INSTRUCTIONS  AND  SUDES 
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US  AF  TEST  PILOT  SCHOOL 


PROGRAM  ASSESSMENT  REVIEW 


The  purpose  of  Program  Assessment  Reviews  at  the  USAF  Test  Pilot  School  is  to 
inform  the  Commandant  and  selected  staff  of  the  status  of  Test  Management  projects.  You 
will  normally  be  scheduled  to  give  detailed  PAR  briefings  at  approximately  the  25%  and  75% 
completion  points  of  the  active  testing  phase,  during  the  weekly  Ops  Meeting.  However,  be 
prepared  each  week  to  present  your  Program  Assessment  Review  or  as  a  minimum  quickly 
discuss  program  status.  PAR  briefings  should  be  approximately  five  minutes  long  using  the 
following  chart/slide  formats: 

1 .  Test  Objectives.  The  test  objectives  should  be  summarized  on  the  slide  shown  in 
Attachment  1  for  those  persons  not  familiar  with  the  project. 

2.  Program  Schedule  Status.  The  Standard  Program  Schedule  Status  format  is  shown  in 
Attachment  2.  Milestones  should  be  essentially  the  same  as  those  in  the  Test  Concept. 
Symbology  will  be  in  accordance  with  Attachment  3. 

3.  Flying  Schedule  Status.  Flight  hour  progress  should  be  discussed  using  the  slide  format 
shown  in  Attachment  4. 

4.  Management  Emphasis.  Use  the  Management  Emphasis  chart  (Attachment  5)  when  ever  a 
program  issue  is  other  than  satisfactory.  Use  more  than  one  chart  if  required  to  clarify  the 
situation(s)  which  has  made  the  program  issue  marginal  or  unsatisfactory. 
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